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Abstract 

We present a convergence theory for pattern search methods for solving bound con- 
strained nonlinear programs. The analysis relies on the abstract structure of pattern 
search methods and an understanding of how the pattern interacts with the bound con- 
straints. This analysis makes it possible to develop pattern search methods for bound 
constrained problems while only slightly restricting the flexibility present in pattern 
search methods for unconstrained problems. We prove global convergence despite the 
fact that pattern search methods do not have explicit information concerning the gra- 
dient and its projection onto the feasible region and consequently are unable to enforce 
explicitly a notion of sufficient feasible decrease. 
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1, Introduction. This paper extends the class of pattern search methods for un- 
constrained minimization, considered in [14], to bound constrained problems: 

, . minimize f(x) 

' ' subject to i < x < u, 

where / : R" -r+ R, £, x , u £ R n , and l < u. We allow the possibility that some of the 
variables are unbounded either above or below by permitting £j,Uj = ±oo, j = 1, . , . , n. 

Our convergence analysis is guided by that for pattern search methods for un- 
constrained problems [14]. We can guarantee that if the objective / is continuously 
differentiable, then a subsequence of the iterates produced by a pattern search method 
for problems with bound constraints converges to a stationary point of problem (1). 
By a stationary point of problem (1) we mean a feasible point x that satisfies the first- 
order necessary condition for optimality: for all feasible z £ R n , (V/(x) , z — x) > 0. 
Equivalently, x is a Karush-Kuhn-Tucker point for problem (1). As in the case of 
unconstrained minimization, pattern search methods for bound constrained problems 
accomplish this without an explicit representation of the gradient or the directional 
derivative. In particular, we prove global convergence in the bound constrained case 
even though pattern search methods do not have explicit information concerning the 
gradient and its projection onto the feasible region and consequently do not explicitly 
enforce a notion of sufficient feasible decrease. 

The general specification of pattern search methods for bound constrained mini- 
mization gives one broad latitude in designing such algorithms. Moreover, as we shall 
discuss, classical pattern search methods for unconstrained minimization — such as co- 
ordinate search with fixed step sizes and the original pattern search of Hooke and Jeeves 
[9] — can be generalized without modification to the bound constrained case. We also will 
show that not all pattern search methods for unconstrained minimization immediately 
generalize to bound constrained problems: in §5.2 we will present a counterexample 
that defeats G.E.P. Box’s method of evolutionary operation using two-level factorial 
designs [1], [3, 12] and show how the convergence theory guides us to a remedy that 
uses composite designs [2], instead of the simpler factorial or fractional factorial designs. 
The multidirectional search algorithm of Dennis and Torczon [7, 13] also requires us to 
augment the pattern used for the algorithm; again we find a straightforward exten- 
sion, but one that reveals much about the interesting behavior of the simplices which 
characterize that method. 

Notation. We denote by R, Q, Z, and N the sets of real, rational, integer, and 
natural numbers, respectively. 

Unless otherwise noted, norms are assumed to be the Euclidean norm. The feasible 
region for problem (1) we denote by 0: 

Q = { x € R 71 | £<x<u}. 
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The projection onto ft we denote by P. If for scalar t we define 

£j if t < lj 

Pj(t) = < t if £j < t < uj 

. uj if t>uj, 

then the projection of x — (xi, • • • , x n ) T is given by 

n 

P(x) = 53 Pi (Xj)ej, 
i = i 

where {e^}, j = 1, • • • , n are the standard basis vectors. On those few occasions where 
we must denote components of subscripted vectors, we use the following notation: qk,j 
denotes the j th component of the vector qk- 

We will denote by g(x) the gradient V/(x) of the objective. Finally, let 

Ln(y) ~ { x € ft J f{x) < f(y ) } . 

2. Pattern Search Methods. We begin by defining the general pattern search 
method for the bound constrained problem (1); it differs from that for unconstrained 
problems [14] in only a few particulars, which we summarize in §2.5. 

2.1. The Pattern. As with pattern search methods for unconstrained problems, 
to define a pattern we need two components: a basis matrix and a generating matrix. 
The basis matrix is a nonsingular matrix B € R nxn . 

The generating matrix is a matrix Ck € Z nxp , where p > 2n. We partition the 
generating matrix into components 

(2) C k = [ M k -M k Lk ] = [ T k L k ]. 

We require that Mk € M C Z nXn , where M is a finite set of nonsingular matrices, and 
that Lk € Z nx( - P ~ 2n '> and that it contains at least one column, a column of zeroes. 

A pattern Pk is then defined by the columns of the matrix Pk = BCk- For conve- 
nience, we use the partition of the generating matrix Ck given in (2) to partition Pk as 
follows: 

P k = BC k = [ BM k -BM k BL k ) = [ BT k BL k ]. 

We also require the matrix BMk to be diagonal: 

(3) BMk = diag(o^), i = 1,, . . , n. 

This condition, absent in the case of unconstrained minimization, is needed in order to 
ensure that we can find feasible points in the pattern that will also produce decrease in 
the objective. As we shall see, this condition is not especially restrictive and is satis- 
fied by all of the commonly encountered pattern search algorithms or straightforward 
variants of them. 

Given A k € R, A k > 0, we define a trial step s\ to be any vector of the form 

s\ = A kBc\, where c\ denotes a column of Ck = [c], •• • c p ]. We call a trial step s\ 

feasible if (xk-\-s\) € ft. At iteration k, a trial pointis any point of the form x\ == Xk+s}., 
where Xk is the current iterate. 
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2.2. The Bound Constrained Exploratory Moves. Pattern search methods 
proceed by conducting a series of exploratory moves about the current iterate Xk to 
choose a new iterate Xk + 1 = Xk + Sk, for some feasible step Sk determined during the 
course of the exploratory moves. The following Hypotheses on Bound Constrained 
Exploratory Moves allows a broad choice of exploratory moves while ensuring the prop- 
erties required to prove convergence. By abuse of notation, if A is a matrix, y € A 
means that the vector y is a column of A. 

Hypotheses on Bound Constrained Exploratory Moves. 

1. Sfc € AkPk = AkBCk = Aft [BTk BLk). 

2 . (xk + Sk) € 0 . 

3. If min{ f (x k + y) | y € A k BT k , x k + y € 0 } < /(«*), 
then f(x k + s k ) < f(x k ). 


2.3. The Generalized Pattern Search Method. Algorithm 1 states the gen- 
eralized pattern search method for minimization with bound constraints. To define a 
particular pattern search method, we must specify the basis matrix B, the generating 
matrix Ck, the bound constrained exploratory moves to be used to produce a feasible 
step Sk, and the algorithms for updating Ck and A*. 

Algorithm 1. The Generalized Pattern Search Method for Bound Constrained Prob- 
lems. 

Let Xq € and Ao > 0 be given. 

For k = 0, 1, • • * , 

a) Compute f(xk)- 

b) Determine a step sk using a bound constrained exploratory moves algorithm. 

c) Compute p k = f(x k ) - f(x k + s k ). 

d) If pk > 0 then Xk+i = Xk + Sk . Otherwise Xk+i == Xk- 

e) Update Ck and A*. 


2.4. The Updates. Algorithm 2 specifies the requirements for updating Afc. The 
aim of the update of A* is to force pk > 0. An iteration with pk > 0 is successful; 
otherwise, the iteration is unsuccessful. Note that to accept a step we only require 
simple, as opposed to sufficient, decrease. 

Algorithm 2. Updating A*. 

Let r € Q, r > 1, and {w 0 ,Wi, • • • ,wl) C Z, w 0 < 0, and w, > 0, i = 1 ,-•■ ,L. Let 
6 = t w °, and X k G A = {r*V • • ,t W l ). 

a) If pk < 0 then Afc +i — OAk. 

b) If pk > 0 then Afc + i = XkAk- 

The conditions on 9 and A ensure that 0 < 9 < 1 and A,- > 1 for all A; € A. Thus, 
if an iteration is successful it may be possible to increase the step length parameter A*, 
but Ak is not allowed to decrease. 
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2.5. Differences between Pattern Search Methods for Unconstrained and 
Bound Constrained Minimization. There are only two additional restrictions re- 
quired of pattern search methods to ensure convergence for the bound constrained case. 

First, we note that pattern search methods are feasible point methods; the search 
begins with a point that satisfies the bounds and maintains feasibility throughout the 
search. This can be seen in Algorithm 1, where we require x 0 £ ft. This requirement 
also appears in the Hypotheses on Bound Constrained Exploratory Moves: if simple 
decrease on the function value at the current iterate can be found among any of the 
feasible trial steps contained in the columns of A kBTk, then the exploratory moves 
must produce a feasible step Sk that also gives simple decrease on the function value at 
the current iterate. 

The second, and more interesting, restriction is that the core pattern BMk must 
be defined by a diagonal matrix. Because the columns of the pattern matrix determine 
the directions of the steps that may be considered, we need to ensure that if we are 
not at a constrained stationary point, we have at least one feasible direction of descent. 
Practically, we must ensure that we have directions that allow us to move parallel to 
the constraints. Requiring BMk to be a diagonal matrix is sufficient. And as we will 
show in §5.2, this requirement is unavoidable. 

We note an equivalence between pattern search methods for bound constrained 
problems and an exact penalization approach to problem (1). Applying a pattern 
search method for problem (1) produces exactly the same iterates as applying such an 

f(*). 

if x £ ft 
otherwise. 

In fact, this is one classical approach used with direct search methods to ensure that 
the iterates produced remain feasible (see, for instance, [10, 11]). In the case of pattern 
search methods this formulation is not simply a conceptual approach; pattern search 
methods are directly applicable to this exact penalty function since .they do not rely 
on derivatives. However, as we will demonstrate in §5.2, this exact penalization ap- 
proach cannot be applied with an arbitrary pattern search method for unconstrained 
minimization; we must require that BMk be diagonal. 

2.6. Results from the Unconstrained Theory. We recall the following results 
from [14], to which we refer the reader for the proofs. The first result indicates one 
sense in which A*, regulates step length. 

LEMMA 2.1 (Lemma 3.1 FROM [14]). There exists a constant > 0, independent 
of k, such that for any trial step s k ^ 0 produced by a generalized pattern search method 
(Algorithm 1) we have || s\ || > (*A k . 

The next result is key to the convergence of pattern search methods. It states that 
the iterates produced by a pattern search method have a rigid algebraic structure. 
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algorithm to the unconstrained problem 

minimize 

where 


( 4 ) 


F(x) 


/(*) 
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Theorem 2.2 (Theorem 3.2 FROM [14]). Any iterate x N produced by a gener- 
alized pattern search method (Algorithm 1) can be expressed in the following form: 

N - 1 

( 5 ) X N = X Q +(p r ™ a -rVB^ AoB J2z k , 

k = 0 

where 

• xq is the initial guess, 

• fi/a. = t, with a,/3 e N and relatively prime, and r is as defined in the 
algorithm for updating Ak (Algorithm 2), 

• r ib and r\jB depend on N, 

• Ao is the initial choice for the step length control parameter, 

• B is the basis matrix, and 

• zk e Z n , k = 0, • • • , N — 1 . 

The last result we recollect says, in conjunction with Lemma 2.1, that if we bound 
the size of the elements of the generating matrix (which is a reasonable thing to do), 
then Ak completely regulates the size of the steps a pattern search method takes. 

LEMMA 2.3 (Lemma 3.6 FROM [14]). If there exists a constant C > 0 such that 
for all k, C > ||c*||., for all i — 1, • ,p, then there exists a constant > 0, independent 

of k, such that for any trial step s\ produced by a generalized pattern search method 
(Algorithm 1) we have Ak > ^*|js).||. 

3. Convergence Theory. We now present the first-order constrained stationary 
point convergence theory for pattern search methods for bound constrained problems. 
We begin by defining, for feasible x, the quantity 

q{x) = P(x -g{x)) - x. 

In the bound constrained theory the quantity q(x ) plays the role of g(x) in the un- 
constrained theory, giving us a continuous measure of how close we are to constrained 
stationarity, as in the theory for methods based explicitly on derivatives (e.g., [6], where 
we got the idea). The following proposition summarizes properties of q that we will 
shortly need, particularly the fact that x is a constrained stationary point for (1) if and 
only if q(x) = 0. While stated for the particular domain 0, the proposition holds for 
any closed convex domain. 

Proposition 3.1. Let x ell. Then 


(6) 

II «(*) II < II g{x) || , 

(7) 

|| «(*) || < || P(g(x)) || . 


and 

(8) x is a stationary point for problem (1) if and only if q{x) — 0. 
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Proof. We need the following properties of the projection: 

(a) If ^ € 0 then ( P(x ) — x, z — P(x)) > 0 for all x € R". 

(b) For all x,y € R n , || P(y) - P( x) || < || y - x ||, 

(c) If T(x) is the tangent cone of fi at x, then 

min { (g(x),v) \ v € T(x), j| v |j < 1 } = - || P(g(x)) || . 

The proofs of (a) and (b) may be found in [8], and the proof of (c) may be found in 
[5] (there is a difference in sign between P(g(x)) here and the quantity Vuf [x) as it is 
defined in the latter reference). 

We first prove (6). If x € fi, then P(x) — x and so from (b), 

II <?(«) II = II P( x ~ ffi x )) ~ P ( x ) II ^ II ( x - 9i x )) ~ x II = II 5'(a ; ) II , 

which is (6). ' 

Next we establish (7). Since P(x—g(x)) — x G T(.r), we obtain from (c) the estimate 

(g( x ) , P(x - g(x)) -x)>- || P{g(x)) || || P(x - g[x)) - x || 


or 

(9) (g(x) , q(x)) > - || P{g(x)) || || q{x) || . 

Meanwhile, from (a) we obtain 

(q{ x ) + 9( x ) , -q( x )) = (P( x - g( x )) ~( x ~ g( x )) , x - P{x - g{x))) > o, 
whence 

(10) (g(x ) , q(x)) < - [I q(x) || 2 . 

Combining (9) and (10) we arrive at 

II q(x) II < || P( 9 (x)) || , 


which is (7). 

Now consider (8). First suppose that x is a stationary point for problem (1). Then 
P(g(x)) = 0, so from (7) it immediately follows that q{x) = 0. Conversely, suppose 
that q{x) = 0. Then P(x — g(x)) = x, so from (a) we have, for all 2 € fi, 

(P(x - g(x)) -(x- g(x )) , z-P(x- g(x))) > 0, 


or 


(g(x) , z-x)> 0. 

But this is nothing other than the first-order necessary condition for optimality for 
problem (1) at x. 0 
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We can now state the first convergence result for the general pattern search method 
for bound constrained minimization. 

THEOREM 3.2. Let Lffxo) be compact and suppose f is continuously differentiable 
on Lq(x 0 ). Let {cct} be the sequence of iterates produced by a generalized pattern search 
method for bound constrained minimization (Algorithm 1). Then 

liminf || q(x k ) || = 0. 

k— f+OO 

The proof of this theorem is given in §4.1, after we have established the necessary 
intermediate results. 

We can strengthen the result given in Theorem 3.2 in the same way that we do in 
the unconstrained case [14]. First, we require the columns of the generating matrix C k 
to remain bounded in norm, i.e., that there exists a constant C > 0 such that for all 
k, C > ||4 1|, for all i = 1 , • • • , p. Second, we replace the original Hypotheses on Bound 
Constrained Exploratory Moves with a stronger version, given below. Third, we require 
that limfc_ +00 A k = 0. All the algorithms described in §5, except multidirectional 
search, satisfy this third condition because of the customary choice of A = {1} = {r 0 }. 
However, it is not necessary to force the steps to be non-increasing. 

Strong Hypotheses on Bound Constrained Exploratory Moves. 

1. s/t € A kPk = A k BC k = A k [BTk BLk). 

2. ( x k + s k ) € O. 

3. If min{ f(x k +y) | y € A k BT k , x k + y € ft } < f(x k ), 

then f(x k + s k ) < min { f(x k + y) | y € A k BT k , x k + y £ 12 }. 

THEOREM 3.3. Let Lq(x o) be compact and suppose f is continuously differentiable 
on Lq(x o). In addition, assume that the columns of the generating matrices are uni- 
formly bounded in norm, that limfc_. +00 A k — 0, and that the generalized pattern search 
method for bound constrained minimization (Algorithm 1) enforces the Strong Hypothe- 
ses on Bound Constrained Exploratory Moves. Then for the sequence of iterates {a:*,} 
produced by the generalized pattern search method for bound constrained minimization, 

lim ||?(a:fc)|| = 0. 

k-*+o o 


The proof will be found in §4.2. 

4. Proof of Theorems 3.2 and 3.3. Given an iterate x k , let g k — g(x k ) and 
q k — q(x k ). Let to denote the following modulus of continuity of g: given x € R n and 
£ > 0, 

w{x, e) = sup { 8 > 0 | || g(y) — g(x) || < e for all y such that || y — x || < 6 } . 

We begin with an elementary proposition concerning descent directions. 
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PROPOSITION 4.1. Let d € R n and suppose that f restricted to the closed line 
segment connecting x and x+d is continuously differentiable. Assume, too, that g(x) 0 
and g{x) T d < —e || d || , Then, if || d || < w(x, |), 

f{x + d)-f(x) < -||| d'||. 

Proof. From the mean-value theorem, we have, for some y on the line segment 
between x and x + d, 

f(x + d) - f{x) = g(x) T d + (g(y) - g(x)) T d 

< -e\\d\\ + \\g(y)-g(x)\\\\d\\. 

If || d || < u>(x, |), then || g(y) — g{x) || < | and the result follows. □ 

In the proof of the next result the bound constrained and the unconstrained cases 
most differ. The proof of Proposition 4.2 implicitly relies on the fact that in the bound 
constrained case, the directions in the pattern defined by the columns of BMk are 
coordinate directions and thus are oriented normal and tangent to the faces of the 
feasible region. That this is not merely convenient is made clear by the counterexample 
in §5.2. 

PROPOSITION 4.2. Suppose that f is continuously differentiable on Lq(x o) and 
q(x k ) 7 ^ 0. Then there exists a Vk > 0 such that if Ak < Vk, then there is a trial steps\ 
defined by a column of AkBTk for which ( Xk + s|.) € fl and 


9k s i < 


% 


4 


5 


where c n = n~ 2 . 

Proof. We restrict our attention to the steps defined by the columns of AkBTk] by 
hypothesis, AkBTk = A kB[Mk —Mk] = A^[diag(dy — diag(dj;.)] (see (3)). Choose an 
index m for which 

(11) I 9k, m I = || Hoc > II II 1 

where qk, m is the m th component of qk. Note that it is also the case that 


( 12 ) 


9k, m ^ 9k, m 


and sign {g k , m ) = sign(g fc , ro ). 

Let si = — sign(<?fc im )AA; | d™ | e m ; this vector will be among the columns of A k BTk- 
Since x k + q k — P(x k gk) is feasible, we have i < x k + q k < u and thus 


i-m S X k ,m d" 9k, m ^ U m . 


It follows that if A k \ d™ \ < \ q k , m |, then the trial point x\ — x k + s\ will be feasible. 
Moreover, from (11) and (12), 


9k 4 = -sign{g k ,m)Ak\ d% \g k , m * 


4 


9k,m 



4 


9k || • 
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Defining Vk = || qk /Id™] then does the trick. 0 

PROPOSITION 4.3. Suppose that Lq(x 0 ) is compact and that f is continuously 
differentiable on Lq(xq). Then given any q > 0, there exists 6 > 0, independent of k, 
such that if A k < $ and |j q(x k ) || > q, the pattern search method for bound constrained 
minimization will find an acceptable step s k ; i.e., f(xk + Sk) < f(xk ) and (xk + s k ) € 0 . 

If in addition, the columns of the generating matrix remain bounded in norm and 
we enforce the Strong Hypotheses on Bound Constrained Exploratory Moves, then, given 
any q > 0, there exist 6 > 0 and a > 0, independent of k, such that if Ak < 6 and 
|| q(x k ) || > q, then 


f(x k +i) < f(x k ) ~ cr II q(x k ) || || s k || . 

Proof. Since g{x) is uniformly continuous on Ln(xo), there exists w* > 0 such that 

u (xk,n~?q) > w* 

for all k for which || qk || > q. 

Next, choose d* > 0 such that d\ < d* for all i and k. This we can do because the 
set { d \ } is finite (see (3) and the conditions on M* given in §2.1). Let 

V 

n 2 a* 

then 



for all k for which || qk || > q, where Vk is as in Proposition 4.2. 
Finally, let 


6 - min(i/*,w */d*). 

Now suppose I] q(xk) || > q and Ak < 8. Since Ak < Vk) Proposition 4.2 assures us of 
the existence of a step s\ defined by a column of AkBTk such that (xk + s l k ) € D and 

9k s k < ~Cn || qk || || 4 | • 

At the same time, we also have 

4 < A k d* < u*<u ( Xk , Cn II qk II) . 

So by Proposition 4.1, 

f(x k + 4) - /(*») < II qt II 1 4 1 . 

Thus, w r hen A* < 8, f(x \ ) = f(x k + 4 ) < f{ x k) for at least one feasible $1 € AkBTk- 
The Hypotheses on Bound Constrained Exploratory Moves guarantee that if 

min { f(x k + y) | y € A k BT k , x k + y G 0 } < f(x k ), 
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then f(xk-\-Sk ) < f(xk ) and (xk + Sk) € 0. This proves the first part of the Proposition. 

If, in addition, we enforce the Strong Hypotheses on Bound Constrained Ex- 
ploratory Moves, then we actually have 


f(x k+1 )-f(x k )<- 1 1| q k 



Lemma 2.1 then ensures that 

f{xk+ 1 ) < f{x k ) - yC*Ajfc || g(x fc ) || . 


Applying Lemma 2.3, we arrive at 


/(**+*) < /(**) - ^ II ?(**) II II II , 


where <r = 0 

COROLLARY 4.4. Suppose that Lq(x o) is compact and that f is continuously dif- 
ferentiable on Lq(x o). Suppose, too, that lim inf^ +00 || q(x k ) || ^ 0. Then there exists 
a constant A* > 0 such that for all k, A k > A* . 

Proof. By hypothesis, there exists N and q > 0 such that for all k > N, || q(x k ) || > 
q. By Proposition 4.3, we can find 6 such that if k > N and A k < 6, then we will find 
an acceptable step. In view of the algorithm for updating A k in §2.4, we are assured 
that for all k > N, Ak > 06. We may then take A* = min{A 0 , • • • , A at, 06}. D 

The next theorem combines the strict algebraic structure of the iterates with the 
simple decrease condition of the generalized pattern search algorithm for bound con- 
strained problems, along with the algorithm for updating A*, to give us a useful fact 
about the limiting behavior of A*. 

THEOREM 4.5. Assume that Lq(x o) is compact. Then lim inffc_+co Afc = 0. 

Proof. The proof is like that of Theorem 3.3 in [14]. Suppose 0 < Alb < A a, for 
all k. Using the algorithm for updating A* found in §2.4, it is possible to write Ak as 
Ak = r rk Ao , where nt € Z : . 

The hypothesis that A lb < Afc for all k means that the sequence {t 7- *} is bounded 
away from zero. Meanwhile, we also know that the sequence {Afc} is bounded above 
because all the iterates Xk must lie inside the set Lq(xq) = {x € 0 : f (x) < f{x o)} and 
the latter set is compact; Lemma 2.1 then guarantees an upper bound A ub for {Afc}. 
This, in turn, means that the sequence {r Tk } is bounded above. Consequently, the 
sequence {r rk } is a finite set. Equivalently, the sequence {rfc} is bounded above and 
below. 

Next we recall the exact identity of the quantities tlb and ruB in Theorem 2.2; the 
details are found in the proof of Theorem 3.3 in [14], In the context of Theorem 2.2, 


riB 


min {rfc} 
o <k<N 1 J 


ruB — max {r k }. 
o <k<N l J 


If, in the matter at hand, we let 


(13) 


r LB = min {r k } 

0<k<+co 


tub 


max 

0<fc<+co 


i r k}, 


10 



then (5) holds for the bounds given in (13), and we see that for all k, x k lies in the 
translated integer lattice G generated by x 0 and the columns of (3 rLB a~ TVB AqB. 

The intersection of the compact set Lq(x 0 ) with the lattice G is finite. Thus, there 
must exist at least one point 2 * in the lattice for which 2 * = 2 * for infinitely many k. 

We now appeal to the simple decrease condition in Algorithm 1 (d), which guar- 
antees that an iterate cannot be revisited infinitely many times since we accept a new 
step Sk if and only if f(x k ) > f(xk + s*) and (xk + s k ) € 0. Thus there exists an N 
such that for all k > N, Xk — 2 *, which implies that p k = 0. 

We now appeal to the algorithm for updating A k (Algorithm 2 (a)) to see that 
Ak — > 0, thus leading to a contradiction. 0 

4.1. The Proof of Theorem 3.2. The proof is like that of Theorem 3.5 in [14]. 
Suppose that lim inf fc_ +co || q{x k ) || 4 1 0- Then Corollary 4.4 tells us that there exists 
A* > 0 such that for all fc, Ak > A*. But this contradicts Theorem 4.5. 

4.2. The Proof of Theorem 3.3. The proof, also by contradiction, follows that 
of Theorem 3.7 in [14]. Suppose limsup^,^ j) q(xk) |j / 0. Let e > 0 be such that 
there exists a subsequence [| q( x mi ) || > e. Since 

liminf || q(x k ) || = 0, 

K— H j-OO 

given any 0 < q < e, there exists an associated subsequence U such that 

|| q(x k ) [J > q for m 2 - < k < || q( 2 /,) || < rj. 

Since A k —■ ► 0, we can appeal to Proposition 4.3 to obtain for m, < k < i sufficiently 

large, 

f(x k )-f(xk+i) > <r || q(x k ) || || s k || > aq || s* || , 
where cr > 0. Then the telescoping sum: 

U 

{f(Xm i )-f(x m ,+i)) + (f{Xm i +l)-f{x mi +2)) + --- + (f(xi i -l)-f{xi i )) > a V II || 

k—mi 

gives us 

f(x mi ) - f(xu) > Efcm, <7V II s k II > C || X mi - x h || . 

Since / is bounded below, f(x mi ) — /( 2 ( ; ) — » 0 as i — > + 00 , so j| x mi — || — > 0 

as i — ^ + 00 . Then, because q is uniformly continuous, |j q(x mi ) — q(xi t ) || < q, for i 

sufficiently large. However, 

(14) || q(x mi ) || < || q{x mi ) - q{x u ) || + || q(x h ) || < 2 q. 

Since (14) must hold for any q, 0 < q < e, we have a contradiction (e.g., try q = |). 
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5. Examples of Pattern Search Methods for Bound Constrained Min- 
imization. A section of [14] is devoted to showing that each of the following four 
algorithms: 

• coordinate search with fixed step lengths, 

• evolutionary operation using two-level factorial designs ([1] and [3, 12]), 

• the original pattern search method of Hooke and Jeeves [9], and 

« the multidirectional search algorithm of Dennis and Torczon ([7] and [13]) 
are pattern search methods for unconstrained minimization. In this section we will 
discuss how these algorithms may be extended to bound constrained problems. We 
shall see that coordinate search and the pattern search method of Hooke and Jeeves 
extend without modification to the bound constrained case. On the other hand, in the 
case of multidirectional search, we must require the initial basis matrix to be a diagonal 
matrix (in the unconstrained case, we can allow any nonsingular basis matrix); in 
addition, we must augment the columns of the generating matrix to ensure a sufficient 
set of search directions. In the case of evolutionary operation, we also must augment 
the columns of the generating matrix, which we do using a classical variant of factorial 
designs [2]. 

The difference between pattern search methods for unconstrained problems and 
bound constrained problems lies in the two additional conditions discussed in §2.5. 
First, pattern search methods for bound constrained problems must start with a feasible 
iterate and choose feasible trial steps. Second, the core pattern BMk must be defined 
by a diagonal matrix. 

We assume that we begin with a feasible iterate; by design pattern search methods 
for bound constrained problems thereafter accept only feasible iterates. Thus, the only 
thing we will really need to check is that the core pattern BMk is defined by a diagonal 
matrix. 

It is this latter condition that causes us to restrict the admissible choice of the 
basis matrix in multidirectional search and then augment the columns of the generat- 
ing matrix. Moreover, G.E.P. Box’s method of evolutionary operation using two-level 
factorial designs does not satisfy this diagonality condition; we will present a simple 
counterexample that show how evolutionary operation can fail as a consequence in the 
bound constrained case and propose a straightforward remedy for the problem. 

5.1. Coordinate Search and the Pattern Search Method of Hooke and 
Jeeves. Coordinate search and the pattern search method of Hooke and Jeeves extend 
to bound constrained problems without change. In both cases the basis matrix B is 
typically chosen to be a diagonal matrix: either the identity or a matrix whose entries 
reflect the relative scaling of the variables. Furthermore, the first 3 n columns of Ck, 
which are fixed for all iterations k of both algorithms, are composed of all possible 
combinations of {—1,0, 1}. In [14] these columns are organized so that the first 2 n 
consist of the identity matrix I and its negative — I. In terms of our formalism, then, 
Mk — I for all iterations k. It follows that BMk is a diagonal matrix, as required. 
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5.2. Evolutionary Operation Using Factorial Design. For the evolutionary 
operation algorithm using factorial designs, the basis matrix B is usually selected to be 
the identity or a diagonal matrix chosen so that the entries along the diagonal represent 
the relative scaling among the variables. However, this convention is not sufficient to 
ensure that BMk is a diagonal matrix. 

The problem lies with the generating matrix C = [M —M L\. (The generating 
matrix C is fixed across all iterations of evolutionary operation.) The generating matrix 
contains in its columns all possible combinations of {—1,1} to which is appended a 
column of zeroes. Clearly, no subset of n columns of C can be chosen to form a diagonal 
matrix M. 

The need for such a requirement in the bound constrained case can be seen from 
the following example. Consider the pattern search method of evolutionary operation 
using two-level factorial designs applied to the problem 

minimize /(x) = — (xi + 2 x 2 ) 
subject to 0 < xi < 1 
x 2 < 0. 

Suppose that we take as our initial iterate the point (0, 0); the pattern for factorial 
design would consist of the points indicated by the open circles in Figure 1 (we show 
a pattern with A* = 1/4). We see that for any choice of A*, < 1 there only ever will 
be one feasible step produced by the pattern: the step given by the point in the lower 
right-hand corner of the pattern— in the figure, the point (1/4, —1/4)— and that this 
step will only produce increase in /. So evolutionary operation will never move from 
its starting point. 

One remedy can be found in related work by G.E.P. Box and K.B. Wilson [2]. This 
would be to use a composite design instead of the two-level factorial design usually 
proposed. An example of such a design — one that satisfies the requirements of the 
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Fig. 2. An illustration of how the ’problem can be circumvented using a composite design. 

bound constrained global convergence theory — chooses M to be the diagonal matrix 
with entries of 2 along the diagonal. With the additional 2 n columns that would then 
augment the original pattern, the algorithm would not fail for the example given above, 
as shown in Fig. 2. We now have a feasible step along the active constraint X 2 < 0 that 
will produce descent for any choice of A* < 1 / 2. 

As noted in our discussion of Proposition 4.2, by requiring BMk to be a diagonal 
matrix, we are guaranteed that the directions in the core pattern are oriented normal and 
tangent to the faces of the feasible region. As our example illustrates, this requirement 
is essential. 

5.3. Multidirectional Search. The reader should be forewarned that our de- 
scription and discussion of multidirectional search takes a point of view that is osten- 
sibly at odds with the formalism of §2.1. The generating matrix T is viewed as fixed; 
typically T = [M — M ] = [I —I]. The basis matrix, on the other hand, is viewed as 
varying from iteration to iteration so that Bk corresponds to the edges in the current 
simplex that are adjacent to the current iterate Xk . This is the reverse of the discussion 
in §2.1, where B is fixed and IT varies. However, the former view of multidirectional 
search is not incompatible with the formalism of pattern search methods, as noted in 
[14], and as we shall have reason to discuss here. 

The extension of multidirectional search to problems with bound constraints re- 
quires us to restrict the choice of a starting simplex and to augment the columns of the 
generating matrix. 

The first restriction is minor and is usually satisfied by the customary choices made 
in practice. In multidirectional search, the columns of B$ are formed from the edges of 
an initial simplex adjacent to the initial iterate Xq. In the case of bound constraints, 
we restrict the starting simplex to be a right-angled simplex, i.e., the n + 1 vertices of 
the simplex are x 0 and the points x 0 + cqe,-, where cq € R and i = 1, • • • , n. Because of 
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this choice, B 0 = diagfa,). Since M = I, the product B 0 M is a diagonal matrix. 

However, even if the initial simplex is restricted to be a right-angled simplex so 
that B 0 M is diagonal, there is no guarantee that in subsequent iterations B k M will be 
diagonal. To understand why this is so, and how this may be corrected by augmenting 
the columns of the generating matrix, we need to discuss how multidirectional search 
fits within the formalism of pattern search methods. These details are absent from [14], 
so we present them here. 

At iteration k , the basis matrix is 

B k =[bl--- bl] = [(u£ - y°) • • • K - v l )} , 

where v l k , i = 0, • • • , n are the vertices of the simplex associated with multidirectional 
search at this iteration. Define 

f / * = 0 

' ~ t - (l ~ e i e I ~ £m=l e *' e m) * = , n. 

Now consider what happens in the next iteration. If the iteration is unsuccessful, then 
u° +1 = and the new basis for the pattern, which is determined by the edges of the 
simplex emanating from u° +1 , is 


Bk+i = B k = B k T 0 . 

If, on the other hand, the iteration is successful, then v k+l = v k — (vj. — y°) for some 
j = 1, . . . , n, and the new basis will be the set of vectors 


In this case, 


*4+i = 


K 


-*4 + K 


if i = j 
otherwise. 


B k +i = B k Tj. 

Thus, in general, 

( 15 ) B k +\ = B k Tj k+1 , 


and so 

(16) B t = B M T h = **- l T*_ i r* = ■■• = 

i= 1 

Our next goal is to simplify this relation further. 

A short calculation shows that for i, £ = 1, . . . , n, 

n n 

TiT e = I - e t ej - ^ e - e,ef + + e + e,-ej, 

771= 1 771 — 1 
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where 8\ is the Kronecker delta. If i = i, this reduces to 


(17) 

and if i ^ l, 

(18) 

(19) 

and 

( 20 ) 


TiTi = I 


T t T ( = I - e e ej - ]T e ? e m ~ e i e I + e t -ej’ — -T t - e t ef + e t ej. 


m = 1 


Tt&i = - ( / - e*ej - X) I e { = e* - 


m— 1 


(2i (-2») «*• - ( -/ + e^ej + ]T e<e£ + e,-ef - e;^ j e; = e e . 


m = 1 


From (17), (18), and (19) we obtain 

TtTiTt = T e (j-Te — e;ef + e,-ej^ 

= —I — T^eJ + 

= -I - [e t - ei)ej + (e e - e t )ej 
(21) = - (i — eief — e?ej + e?ef + e,ej) . 


This latter matrix we recognize as the negative of the elementary permutation matrix 
E(i , £) that swaps the i th and £ th columns when acting on matrices from the right. Using 
(17) and (21), we obtain the rule 


TiT e = 


I if i = £ 
—Tf_E(i 1 £) otherwise. 


We can then use this formula to reduce (16) to 

B k — ±B 0 Te k lik, 

for some T4 and permutation matrix life. 

This relationship reveals several things. The first is that it reconciles the usual 
description of multidirectional search with the formal abstract definition of a pattern 
search method; the pattern matrix is given by 

(22) B k C w ±B 0 TMI -I 0] = B 0 [T jk - T jk 0]II fe = BC k . 


That is, we may interpret multidirectional search in terms of a fixed basis B and a 
changing generating matrix C k - 
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We can also see that while BT 0 will be diagonal, this diagonality may be lost in 
subsequent iterations. However, the form of the generic pattern from the unconstrained 
algorithm suggests one way to circumvent this problem in the bound constrained case. 
This remedy will, moreover, preserve the geometric interpretation of the pattern in 
multidirectional search in terms of a simplex. 

First, if we ignore the permutation in (22), which only affects column ordering, the 
pattern at iteration k in the unconstrained case is given by 

B k C = BC k = B 0 [T jk - Tj k 0]. 

Suppose we augment the columns of C to include all the 7): 

C = [-T 0 -Ti ••• —T n 0 ]. 

At any iteration k , up to a column permutation, the basis matrix is the matrix B k = 
±BTj k , j k £ {0, • - * , n}. When we then form the pattern P k = A k B k C, we have 

P k = A k B k C = A k B[±T jk T 0 ±T jk T x • • • ±T jk T n ] = A k BC k . 

Now note that (20) means that for j k / l the jf 1 column of —Tj k Tt is the 7 th basis 
vector. Consequently, we are guaranteed that by a permutation of the columns of C k , 

c k = [/ —I L k ) = [r ik], 

where L k changes at each iteration, but V does not. Since we require the initial simplex 
to be a right-angled simplex, we may then be assured that BT = [diag(cq) — diag(cq)], 
as required. 

Moreover, this augmentation of C and the search through its columns can be imple- 
mented in a way that preserves the relationship of the pattern to the moving simplex 
that characterizes multidirectional search. This is possible because the matrices T 
i = 0, • • • , n capture how the basis changes in association with a change of simplex. 
This is the gist of (15). The implications for any implementation of this modification 
to multidirectional search to handle bound constraints will appear elsewhere. 

6. Conclusion. We have presented a reasonable extension of pattern search meth- 
ods for unconstrained minimization to bound constrained problems. The extension is 
supported by a global convergence theory as strong as that for the unconstrained case. 
The generalization makes few additional requirements and as we have seen in §5, the 
classical pattern search methods for unconstrained minimization or straightforward vari- 
ants thereof carry over to the bound constrained case. 

The extension to bound constrained problems also points the way to handling 
general linear inequalities. General linear inequalities will require a sufficiently rich set 
of directions in the pattern so that at any face of the feasible region, one will have 
directions both normal and tangent to the constraints. This we will pursue elsewhere. 

One issue we have not discussed is that of identifying active constraints, as in 
[4, 5]. One would wish to show that if the sequence {a:*;} converges to a nondegenerate 
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stationary point x*, then in a finite number of iterations the iterates Xk land on the 
constraints active at x* and remain thereafter on those constraints. 

There are three difficulties in proving such a result for pattern search methods for 
bound constrained minimization. The first is relatively minor. If the iterates x* are 
to identify the active constraints for a stationary point on the boundary of the feasible 
region, we must ensure that the lattice manifest in Theorem 2.2 actually allows iterates 
to land on the boundary. This requires additional but straightforward conditions on 
such quantities as x 0 , r, A 0 , and the pattern matrices P*. A related but more subtle 
difficulty is that the relative sizes of the steps in the core pattern and the remaining 
points in the pattern must obey certain relations in order to ensure that the algorithm 
does not take a purely interior approach to a point on the boundary. This rules out, for 
instance, certain of the composite designs suggested by G.E.P. Box and K.B. Wilson 
[ 2 ]. 

The most serious obstacle, which remains to be overcome, is showing that ultimately 
the iterates will land on the active constraints and remain there. For algorithms such as 
those considered in [4, 5], this is not a problem because the explicit use of the gradient 
impels the iterates to do this in the neighborhood of a nondegenerate stationary point. 
However, pattern search methods do not have this information. On the other hand, the 
kinship of pattern search methods and gradient projection methods makes us hopeful 
that ultimately we will be able to prove that pattern search methods also identify the 
active constraints in a finite number of iterations. 
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